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represents  natural  learning  situations  by  including:  (^1^  instructions  and 
rewards  that  emphasize  gradual  development  of  understanding,  rather  than 
discovery  of  Mthe  right  rule?**  and  ($1  a  large  number  of  cues,  which  must 
be  discovered,  rather  than  a  few  cues  explicitly  given.  Results  with  12 
college-student  subjects  indicate  significant  learning  in  a  computer-displayed 
task,  over  approximately  10  hours  of  experience.  Learning  was  incremental, 
and  was  accompanied  by  the  addition  of  valid  factors  to  existing  rules-  These 
results  contrast  with  findings  that  people  fail  to  utilize  information  effec¬ 
tively  in  probabilistic  situations.  Earlier  studies  do  not,  however,  model 
situations  in  which  learning  requires  the  discovery  and  validation  of  predic¬ 
tive  cues,  processes  critical  for  the  development  of  real-world  expertise. 
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Learning  in  a  Probabilistic  Environment:  A  New 
Approach*  and  Some  Preliminary  Findings 

Planning  next  year's  budget,  deciding  when  to  plant  your  corn,  selecting 
a  class  of  graduate  students,  •  •  •  What  these  activities  have  in  common  Is 
that  they  all  require  us  to  predict  the  behavior  of  complex,  multifactor, 
probabilistic  environments.  Indeed,  we  face  this  task  whenever  we  must  deal 
with  the  economy,  the  weather,  or  almost  any  aspect  of  human  behavior.  Even 
the  behavior  of  purely  mechanical  systems  is  effectively  probabilistic  to 
those  of  us  with  imperfect  knowledge  (consider  the  vagaries  of  the  family 
car)  • 

The  research  discussed  here  is  concerned  with  the  question  of  how  people 
come  to  understand  such  systems.  Specifically,  how  do  people  learn  the 
relationships  between  factors  in  the  environment  when  those  relationships  are 
"Imperfect,"  that  Is,  correlational  rather  than  strictly  lawful?  Given  that 
we  must  operate  In  a  probabilistic  world,  this  learning  process  Is  essential 
for  the  development  of  real-world  expertise. 

There  is,  of  course,  already  a  long  history  of  research  on  this  general 
topic,  under  the  rubric  of  "probability  learning"  (see  reviews  by  Brehmer, 
1980;  Hammond,  Stewart,  Brehmer,  &  St e Inman,  1975;  Slovlc  &  Lichtenstein, 
1971).  There  have  been  many  variations  in  30  years  of  this  research,  but 
there  has  been  a  common  basic  paradigm.  The  subject's  task  is  to  predict  a 
criterion  value  (e.g.,  length  of  a  line)  based  on  some  predictive  variables 
("cues").  Most  often,  the  cue  variables  are  given  arbitrary  labels  (e.g., 

A,  B,  C)9  and  the  subject  receives  numerical  information  on  each  cue  (e.g., 

"A  ■  4,  ■  6,  £  •  1).  After  receiving  this  information,  the  subject  makes  a 

prediction,  and  subsequently  is  shown  the  true  outcome  (e.g.,  the  line 
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associated  with  [4,  6,  1]).  The  true  outcome  is  a  lawful  function  of  A,  Bf 
and  C,  plus  some  amount  of  random  error*  For  example,  several  studies  have 
used  the  rule 


Y  «  *8sin  x1  +  .4sin  x^  +  *2sin  x^  +  e 

where  Y  is  the  criterion,  and  e  is  a  random  number,  accounting  for 
anywhere  from  12%  to  75%  of  the  variance  in  the  criterion  (Dean,  Hammond,  & 
Summers,  1972;  Hammond,  1971;  Hammond,  Summers,  &  Dean,  1973;  Hoffman,  Earle, 

&  Slovic,  1981)* 

Investigators  using  this  paradigm  are  looking  at  people’s  ability  to 
learn  the  relationships  between  cues  and  criterion  in  the  presence  of 
Imperfect  (i*e*,  probabilistic)  feedback.  To  summarize  30  years  of  research 
very  succinctly,  people  seem  to  be  absolutely  terrible  at  doing  this. 

Consider,  for  example,  the  study  by  Hoffman,  et  al. ,  using  the  three-cue 
function  described  above,  with  12%  random  variation.  Using  the  optimal 
combination  of  factors,  subjects  could  in  theory  achieve  a  correlation  of  .94 
between  their  predictions  and  the  true  outcomes.  After  200  ”stimulus- 
response-outcome”  feedback  trials,  however,  the  average  subject  had  achieved  a 
correlation  of  .21. 

In  his  recent  review  of  probability-learning  studies  (including  many  of 

his  own)  Bernot  Brehmer  concludes: 

People  do  not  learn  optimal  strategies  from  experience  even  if 
the  they  are  given  massive  amounts  of  practice.  •  •  •  This  is 
due  to  lack  of  adequate  schemata  for  handling  the  probabilistic 
aspect  of  the  world.  (1980,  pp.  233-35) 

Subjects  seem  to  be  unable  to  separate  '’signal”  from  ”noise;M  they  reject 

correct  hypotheses  about  relationships,  and  frequently  revive  rejected 

hypotheses.  They  do  seem  able  to  apply  information  given  them,  e.g.,  if  the 

experimenter  informs  the  subject  of  the  relevant  cue-criterion  functions 


HP"W’ 
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(e.g.,  Deane,  Hammond,  &  Summers,  1972;  Hoffman,  et  al.,  1981).  However,  they 
seem  resistant  to  any  attempts  to  help  them  find  the  relationships,  through 
instruction  or  through  structuring  of  feedback  information  (Brehmer  & 
Kuylenstierna,  1978,  1980;  Hoffman  et  al.,  1981). 

Findings  like  these  do  not  bode  veil  for  people's  ability  to  develop  any 
new  understanding  of  their  environments.  They  are  also  troubling,  though, 
because  they  seem  to  contradict  common  everyday  experiences  of  learning. 
Imagine,  for  example,  that  you  ask  about  a  colleague's  whereabouts,  and  are 
told,  "He  usually  stays  home  on  Fridays,  especially  if  the  weather's  nice, 
although  toward  the  end  of  the  term  he's  more  likely  to  be  around.**  A 
statement  like  this  is  not  extraordinary,  yet  it  expresses  a  three-factor 
probabilistic  prediction  rule.  How  can  this  be?  One  possibility  is  that  we 
only  think  we  have  learned  such  rules,  but  they  are  not,  in  fact,  valid  (c.f* 
Einhom  &  Hogarth,  1978  on  "persistence  of  the  illusion  of  validity"). 

However,  it  is  also  possible  that  the  usual  probability-learning  task  misses 
something  Important  about  human  learning  processes — something  that  does  permit 
effective  learning  in  natural  probabilistic  environments. 

To  explore  this  latter  hypothesis,  consider  several  Important  ways  in 
which  the  laboratory  learning  task  may  differ  from  "real-world"  learning 
tasks: 

1.  Linearity  of  cues.  Host  probability-learning  tasks  include  cues 
which  have  a  non-linear,  or  even  non-monotlc,  relationship  to  the  criterion 
(e.g.,  the  three-cue  function  desscrlbed  earlier).  These  relationships  are 
particularly  difficult  to  learn  (Brehmer,  1980;  Hammond  &  Summers,  1965),  but 
it  may  be  easy  to  avoid  these  difficulties  in  natural  settings.  A  number  of 
studies  have  demonstrated  that  complex  systems  can  generally  be  well  modeled 
with  strictly  linear  rules.  Even  when  the  relative  weights  are  "improper,"  or 
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the  true  rule  contains  nonllnearltles,  linear  models  can  often  account  for  a 
high  proportion  of  the  variance  (Daves  and  Corrigan,  1974;  Elnhorn  and 
McCoach,  1977;  Yntema  and  Torgeson,  1961).  This  Is  especially  so  if  there  are 
many,  partially  redundant  cues  in  the  system*  Indeed,  the  one  bright  spot  in 
the  probability-learning  literature  is  that  people  seem  to  be  fairly  good  at 
learning  linear  rules,  even  in  the  presence  of  noise  (e*g*,  Brehmer  & 
Kuylenstlerna,  1978,  1980;  Dean  et  al*,  1972;  Naylor  &  Domine,  1981).  Brehmer 
and  Kuylenstlerna  (1978),  for  example,  used  a  task  with  two  cues  each  having  a 
positive  linear  relationship  to  the  criterion*  The  maxi urn  attainable 
correlation  was  *80,  and  subjects  achieved  a  correlation  above  *70  after  60 
trials*  Thus,  even  if  humans  thought  only  in  terms  of  linear  relationships, 
this  would  still  permit  a  good  deal  of  predictive  ability  in  many  situations* 
2*  Number  and  expllctness  of  cues*  The  typical  laboratory  task  involves 
only  a  very  small  number  of  cues  (usually  1  to  3) ,  and  these  are  explicitly 
identified*  In  natural  situations,  though,  there  are  often  many  possible 
cues,  and  almost  always  an  opportunity  to  discover  and  Incorporate  new 
information*  Building  a  model  of  an  environment,  then,  Involves  two  basic 
processes:  finding  the  cues,  and  figuring  out  how  to  aggregate  them* 

Probability- learning  tasks  eliminate  the  cue-finding  process*  Research 
suggests  that  the  aggregation  process  is  especially  difficult  (e.g*,  Daves, 
1971;  Goldberg,  1970),  and  that  finding  the  cues  may  be  much  more  important* 
Leaving  out  a  variable  is  more  serious  than  misweightlng  it;  thus  Dawes* 
prescription  that  to  build  a  good  (if  not  "optimal")  model,  Mthe  whole  trick 
is  to  know  what  variables  to  look  at  and  then  know  how  to  add**  (Dawes  & 
Corrigan,  1974,  p*  103;  see  also  Dawes,  1979;  Elnhorn  &  Hogarth,  1975)* 

3*  Instructions  and  rewards*  In  the  usual  laboratory  task,  the  gist  of 
the  Instructions  is  to  Nflnd  the  right  rule**  or  **best  rule***  There  is,  then. 
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an  implied  dichotomy  between  “right**  and  "wrong"  rules*  This  is  reinforced  by 
a  reward  system  in  which  the  principal  payoff  for  the  subject  is  the  discovery 
of  "the  rule*"  Furthermore,  in  many  tasks,  until  the  rule  is  found,  little 
achievement  is  possible*  Thus,  there  is  little  reason  to  retain  hypotheses 
which  seem  less  than  perfect,  and  little  opportunity  to  build  upon  partial 
knowledge*  In  contrast,  in  natural  situations,  predictive  models  are  typic¬ 
ally  better  or  worse  overall,  in  a  more  or  less  continuous  way.  Improvements 
in  understanding  are  more  likely  to  be  gradual  or  incremental,  and  reward 
tends  to  vary  continuously  with  predictive  accuracy. 

4.  Time.  In  these  tasks,  the  time  allotted  for  learning  has  been 
extensive  by  laboratory  standards  (several  hours),  but  very  short  in 
comparison  with  the  time-span  of  experience  usually  associated  with  the 
development  of  real-world  expertise. 

The  goal  of  the  research  presented  here  was  to  look  at  learning  processes 
in  a  more  natural  environment,  according  to  the  four  points  described  above. 
That  is,  the  task  tested  here:  (a)  can  be  understood  in  terms  of  linear  cue- 
criterion  relationships;  (b)  provides  many  possible  cues,  not  all  of  which  are 
explicitly  specified;  (c)  Includes  instructions  which  emphasize  Improvement, 
rather  than  ultimate  solution,  and  payoffs  that  vary  continuously  with 
predictive  accuracy;  and  (d)  allows  subjects  adequate  amounts  of  time  for 
learning* 

It  is  hypothesized  that  in  an  environment  like  this,  significant  learning 
will  take  place*  Gradual  Improvement  is  expected,  as  learners  discover  and 
test  new  valid  predictive  cues,  and  add  these  to  their  rule.  As  the  learner’s 
rule  becomes  more  complete,  better  prediction  is  possible.  This  process  of 
addition  of  valid  factors  is  hypothesized  to  be  the  major  means  by  which 
predictive  accuracy  is  improved*  However,  several  other  processes  may  also 
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contribute:  Invalid  factors  mistakenly  included  in  the  model  may  be  expunged; 
weak  cues  may  be  replaced  with  related  cues  that  are  more  directly  predictive; 
and  a  more  precise  understanding  of  the  shape  and  magnitude  of  the  cue- 
criterion  relationships  may  be  achieved.  Note  that  only  the  last  of  these 
processes  is  tapped  in  the  typical  probability-learning  task. 

Methods 

Subjects  interact  with  a  computer  display  by  means  of  a  keyboard.  The 
screen  displays  geometric  figures  varying  in  size,  shape,  line-pattern  (e.g., 
striped,  checkered,  etc.),  and  location.  Around  each  figure  is  marked  a 
circular  "area  of  influence,"  visible  to  the  subject.  In  this  environment  of 
figures,  the  path  of  a  point  is  "traced"  from  a  visible  starting  location,  in 
a  straight  line  in  any  direction  (see  Figure  1).  Subjects  are  told  that  "were 
it  not  for  the  figures,  the  point  would  continue  off  the  screen  in  a  straight 
line,"  but  that  "if  a  trace  touches  the  area  of  influence  around  the  figure, 
the  figure  may  affect  the  trace  by  causing  it  to  stop  somewhere  on  the  screen, 
as  shown  by  a  little  asterisk."  It  is  then  explained  that 

The  object  of  the  game  is  to  predict  where  the  trace  will 
stop,  or  if  it  will  go  off  the  screen.  You  should  understand 
that  this  will  be  difficult,  and  you  are  not  expected  to  be 
able  to  "solve"  it  exactly.  Rather,  you  should  try  and  figure 
out  as  much  as  you  can  about  how  it  works,  so  you  can  make  the 
best  predictions  you  can. 

Twelve  college-student  subjects  participated  in  this  study.  Each  subject 
received  two  types  of  experience  with  the  system:  learning  and  testing. 
Learning  sessions  were  30-mlnute  periods  in  which  subjects  could  freely  design 
their  own  screens  and  conduct  their  own  tests.  They  could  draw  figures  of  any 
of  three  sizes,  three  shapes,  and  three  patterns,  and  place  them  anywhere  on 
the  screen.  They  could  trace  points  starting  anywhere,  and  going  in  any 
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direction*  they  were  free  to  experiment,  observe,  calculate,  and  take  notes 
for  as  much  of  the  30-minute  period  as  they  liked*  Then,  they  went  on  to  a 
testing  phase*  Here,  they  observed  a  set  of  16  screens  representing  a  random 
sailing  of  situations  in  which  a  point  passes  close  enough  to  a  figure  to  be 
(possibly)  affected*  In  each  test  trial  they  were  shown  what  event  would  be 
tried,  and  they  made  a  prediction  as  to  the  outcome,  indicating  whether  they 
believed  the  point  would  stop,  and  if  so,  where*  After  their  prediction,  the 
true  outcome  was  observed*  The  testing  sessions  were  only  about  one-third  the 
length  of  the  learning  sessions  and  new  trials  proceeded  quickly.  Thus,  most 
learning  took  place  in  lerning  sessions,  despite  feedback  during  tests* 

Learning  and  testing  sessions  were  alternated,  with  two  of  each  on  each 
of  seven  days  (about  10  hours  of  experience  with  the  system).  During  this 
time,  the  subjects  were  paid  according  to  the  number  of  points  they  achieved 
in  the  testing  session*  Points  were  awarded  according  to  the  closeness  of 
their  predictions  to  the  true  observed  stopping  point  of  each  test  trace* 

The  true  rule  underlying  the  behavior  of  the  system  was  a  linear 
combination  of  six  cues:  Shape  of  figure;  closeness  of  approach  of  trace  to 
figure;  direction  of  treace  toward  right  or  left;  size  of  figure;  distance 
from  trace  origin  to  figure;  closeness  of  figure  to  center  of  screen*  These 
cues  were  weighted  such  that  each  of  the  first  three  accounted  for  roughly 
twice  as  much  variance  as  each  of  the  last  three*  Note  also  that  only  two  of 
these  cues  were  directly  specified  in  the  display  (size  and  shape  of 
figure)*  The  other  four  had  to  be  discovered  among  the  plethora  of  possible 
spatial  relationships  existing  in  the  environment*  Note  also  that  one  very 
salient  cue,  the  line-pattern  of  the  figure,  was  a  false  cue  in  this  case* 

The  subjects  were  divided  randomly  into  two  conditions,  six  in  each*  In 
the  protocol  condition,  subjects  were  asked  to  "think  out  loud**  during  the 
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procedure,  and  were  alao  questioned  about  their  thinking  at  various  points* 
Those  in  the  non-protocol  condition  were  not  asked  for  any  verbal  responses, 
although  an  experimenter  was  present  to  help  operate  the  computer,  and  to 
handle  any  problems* 


Results 

Based  on  the  model  of  learning  proposed  earlier,  the  main  expectation  was 
that  learning  would  take  place  gradually.  Learning  should  be  incremental,  as 
subjects  discover  and  test  new  valid  predictive  cues,  and  add  these  to  their 
predictive  models* 

Figure  2  shows  that  gradual  improvement  was  indeed  observed,  at  least 
through  the  sixth  session*  The  results  were  analyzed  using  an  ANOVA  with  one 
between-subjects  factor  (condition:  protocol/no-protocol)  and  two  within 
(session:  one  to  seven;  half:  first  test  of  the  day /second  test).  The  im¬ 
provement  with  sessions  was  highly  significant  (F[6,  60]  «  8.78,  p  <  .^001), 
and  no  other  effects  were  significant. 

There  are  also  some  data  about  the  processes  through  which  lemming  was 
accomplished*  One  of  the  responses  required  of  the  six  verbal-protocol 
subjects  was  to  provide  written  "hints"  after  each  day's  experience.  Their 
instructions  were  to  provide  as  many  clues  about  the  system  as  they  could,  as 
though  to  a  naive  participant  whom  they  wanted  to  help  master  the  game. 
Subjects  were  encouraged  to  Include  any  information  they  though  right  help, 
even  if  they  were  not  yet  sure* 

The  hints  were  categorized  according  to  the  nature  of  the  predictive  cues 
they  utilized*  Correct  cues  were  those  which  corresponded  to  one  of  the  six 
valid  cues  in  the  model*  Partly  correct  cues  were  those  which  captured  some, 
but  not  all,  of  a  correct  cue-criterion  relationship  (e.g.,  a  cue  which  was 
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positively  correlated  with  a  correct  cue).  Incorrect  cues  were  cues  which  had 
little  or  no  predictive  value  in  the  environment.  The  most  common  example  was 
a  belief  that  figure-pattern  mattered.  Also,  any  postulated  interactions 
between  cues  were  scored  as  incorrect. 

Figure  3  shows  the  changes  in  these  categories  of  cues  across  the  seven 
days  of  experience.  It  was  hypothesized  earlier  that  the  principle  mechanism 
of  change  would  be  the  addition  of  new,  valid  cues  to  the  model,  and  this  is 
supported  by  the  data  from  the  helpful-hints  reports.  The  number  of  partly- 
correct  cues  seems  to  remain  constant,  but  this  is  the  net  result  of  two 
processes.  New  partially-valid  cues  are  being  discovered  throughout  the 
process,  but  partially-valid  cues  are  also  being  replaced  with  stronger, 
correct  cues.  Finally,  it  is  interesting  that  the  role  of  incorrect  cues  is 
relatively  small  here.  This  is  so  despite  the  fact  that  cues  were  scored  as 
remaining  in  the  subject's  model  until  explicitly  discounted  or  until  an 
incompatible  new  hypothesis  was  expressed.  In  some  cases,  incorrect  cues 
persisted  in  subject's  models,  but  in  many  cases  there  were  a  series  of 
different  incorrect  cues  (e.g..  Interactions)  with  only  brief  tenures. 

The  helpful-hints  results  are  not  definitive,  of  course,  but  they  do 
proovide  support  for  the  hypothesis  that  addition  of  cues  is  a  primary  source 
of  learning,  with  replacement  of  weak  cues  and  removal  of  invalid  cues  as 
secondary  processes.  In  their  comments,  subjects  very  seldom  expressed  any 
quantitative  relationships.  Expressions  of  rules  were  almost  always  ordinal, 
e.g.,  "the  bigger  it  is,  the  sooner  it  stops.**  It  was  rare  even  for  subjects 
to  say  anything  about  the  relative  importance  of  different  cues.  Thus,  there 
is  little  evidence  of  attention  to  cue  weights,  or  to  the  shape  of  the  cue- 
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Conclusions 

This  study  is  clearly  just  a  beginning,  but  it  demonstrates  the  need  for 
new  consideration  of  processes  of  learning  in  probabilistic  environments.  The 
focus  should  be  on  adding,  revising,  and  eliminating  cu^*?  rather  than  on 
pinpointing  the  cue-criterion  function.  There  are  a  great  variety  of 
interesting  question  for  further  research  along  these  lines.  For  example: 

(a)  Is  it  important  that  subjects  be  allowed  to  experiment,  rather  than 
just  observe?  Hoffman  et  al.  (1981)  found  that  this  made  no  difference  in  a 
typical  probability- learning  task,  but  it  might  be  important  in  discovering 
and  validating  new  cues. 

(b)  This  particular  task  was  not  deterministically  predictable  from  the 
subject's  point  of  view.  There  were  always  unknown  controlling  factors,  but 
there  was  no  explicitly  random  element.  There  are  many  important  issues 
concerning  what  " random"  means  (see,  e.g.,  Lopes,  1982).  Suffice  it  to  say 
here  that  the  present  experiment  does  not  contain  any  factors  which  vary 
unpredictably  with  time.  This  may  or  may  not  prove  to  be  important  in  the 
ability  to  learn  from  experience.  Perhaps  for  learners  not  all  unpredict¬ 
ability  is  equal. 

(c)  What  would  tappen  with  additional  learning  time?  Most  of  the  sub¬ 
jects  in  the  present  experiment  were  still  improving  at  the  last  session,  and 
in  all  cases  there  was  considerable  room  for  further  improvement,  it  is  pos¬ 
sible  that  different  learning  processes  may  play  important  roles  in  the  longer 
term.  For  example,  replacement  of  weak  cues  and  attention  to  the  shape  of  the 
cue-criterion  function  might  be  more  prominent  in  later  stages  of  learning. 

(d)  What  is  the  effect  of  an  initial  knowledge  base?  In  the  present 
task,  as  in  most  learning  tasks,  the  subject  starts  with  very  little  knowledge 
of  the  workings  of  the  system.  Natural  learning  situations  provide  varying 
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amounts  of  initial  knowledge  from  prior  experience  and  various  kinds  of  social 
transmission.  How  is  such  information  applied  in  new  learning  situations? 

And  what  happens  in  the  presence  of  false,  misleading,  or  outdated  initial 
information? 

These  questions,  and  many  others  of  equal  interest,  arise  from  a  focus  on 
the  learner's  construction  of  a  predictive  model,  cue  by  cue.  It  is  proposed 
that  these  constructive  processes  are  central  to  the  ability  to  learn  from 
experience  in  complex  probabilistic  environments.  Certainly,  much  of  what  we 
know  comes  from  learners  of  the  past.  The  ability  to  learn  from  experience, 
though,  is  critical  for  understanding  and  controlling  new  environments,  and 
for  going  beyond  what  is  already  known.  In  studying  the  construction  and 
revision  of  predictive  models  during  learning,  then,  we  are  looking  Into  a 
critical  element  in  the  development  of  expertise. 


V 
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FIGURE  CAPTIONS 

Figure  1  •  Be  ample  of  a  display  screen  used  in  this  study  •  A  point  is 
traced  from  a  starting  location  (A)  •  The  point's  behavior  is  affected  by  a 
geometric  figure  (B)  if  it  comes  within  a  close  enough  range  (indicated  by  the 
circumscribed  circle)*  Jh  that  case,  it  may  stop  before  reaching  the  edge  of 
the  screen  (C)  •  Accept  for  the  letters  A,  B,  C,  all  aspects  of  the  display 
were  visible  to  the  subject*  (Adapted  from  Mynatt,  Doherty ,  and  Tweney,  1978, 
with  the  help  of  Don  N  •  KLeinauntz  •) 

Figure  2*  Average  total  test  score  per  subject  (n  *  12),  by  days  of 
experience*  Each  "day"  consisted  of  up  to  one  hour  of  learning  trials,  and 
one-half  hour  of  test  trials.  Maximum  possible  score  is  approximately  750* 
Figure  3*  Changes  in  constituents  of  subjects'  predictive  models 
(n  -  6),  over  days  of  experience*  The  optimal  model  contained  six  correct 


cues  • 


17 


SESSION 


19 


OFFICE  OF  NAVAL  RESEARCH 
Engineering  Psychology  Group 
TECHNICAL  REPORTS  DISTRIBUTION  LIST 


OSD 

CAPT  Paul  R.  Chatelier 
Office  of  Che  Deputy  Under  Secretary 
of  Defense 
OUSDRE  (E&LS) 

Pentagon,  Room  3D  129 
Washington,  D.  C.  20301 

Dr.  Dennis  Leedom 

Office  of  the  Deputy  Under  Secretary 
of  Defense  (ul) 

Pentagon 

Washington,  D.  C.  20301 

Department  of  the  Navy 

Engineering  Psychology  Group 
Office  of  Naval  Research 
Code  442  EP 

Arlington,  VA  22217  (2  cys.) 

Aviation  &  Aerospace  Technology 
Programs 
Code  210 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Communication  &  Computer  Technology 
Programs 
Code  240 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Physiology  &  Neuro  Biology  Programs 
Code  441NB 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 


Deparment  of  the  Navy 

Tactical  Development  4  Evaluation 
Support  Programs 
Code  230 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Manpower,  Personnel  &  Training 
Programs 
Code  270 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Mathematics  Group  • 

Coda  411-MA 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Statistics  and  Probability  Group 
Code  411-S&P 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Information  Sciences  Division 
Code  433 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  2217 

CDR  K.  Hull 
Code  230B 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 


Special  Assistant  for  Marin*  Corps 
Matters 
Code  100M 

Office  of  Naval  Research 
800  North  Quincy  Street 
Arlington,  VA  22217 

Dr.  J.  Lester 
ONR  Detachment 
495  Summer  Street 
Boston,  MA  02210 

Mr.  R.  Lawson 
ONR  Detachment 
1030  East  Green  Street 
Pasadena,  CA  91106 

CDR  James  Offutt,  Of ficer-in-Charge 
ONR  Detachment 
1030  East  Green  Street 
Pasadena,  CA  91106 

Director 

Naval  Research  Laboratory 
Technical  Information  Division 
Code  2627 

Washington,  D.  C.  20375 

Dr.  Michael  Melich 
Communications  Sciences  Division 
Code  7500 

Naval  Research  Laboratory 
Washington,  D.  C.  20375 

Dr.  J.  S.  Lawson 

Naval  Electronic  Systems  Command 
NELEX-06T 

Washington,  D.  C.  20360 

Dr.  Robert  E.  Conley 

Office  of  Chief  of  Naval  Operations 

Command  and  Control 

0P-094H 

Washington,  D.  C.  20350 

CDR  Thomas  Berghage 

Naval  Health  Research  Center 

San  Diego,  CA  92152 


Dr.  Robert  G.  Smith 
Office  of  the  Chief  of  Naval 
Operations,  0P987H 
Personnel  Logistics  Plans 
Washington,  D.  C.  20350 

Dr.  Andrew  Rechnitzer 
Office  of  the  Chief  of  Naval 
Operations,  OP  952F 
Naval  Oceanography  Division 
Washington,  D.  C.  20350 

Combat  Control  Systems  Department 
Code  35 

Neval  Underwater  Systems  Center 
Newport,  RI  02840 

Human  Factors  Department 
Code  N-71 

Naval  Training  Equipment  Center 
Orlando,  FL  32813 

Dr.  Alfred  F,  Smode 
Training  Analysis  and  Evaluation 
Group 

Orlando ,  FL  328  L3 

CDR  Norman  E.  Lane 
Code  N-7A 

Naval  Training  Equipment  Center 
Orlando,  FL  32813 

Dr.  Gary  Poock 

Operations  Research  Department 
Naval  Postgraduate  School 
Monterey,  CA  93940 

Dean  of  Research  Administration 
Naval  Postgraduate  School 
Monterey,  CA  93940 

Mr.  H.  Talkington 
Ocean  Engineering  Department 
Naval  Ocean  Systems  Center 
San  Diego,  CA  92152 


Department  of  Che  Navy 


Mr.  Paul  Heckman 

Naval  Ocean  Systems  Center 

San  Diego,  CA  92152 

Dr.  Ross  Pepper 

Naval  Ocean  Systems  Cancer 

Hawaii  Laboratory 

P.  0.  Box  997 

Kailua,  HI  96734 

Dr.  A.  L.  Slafkosky 
Scientific  Advisor 
Commandant  of  the  Marine  Corps 
Code  RD-1 

Washington,  D.  C.  20380 

Dr.  L.  Chxnura 

Naval  Research  Laboratory 

Code  7592 

Computer  Sciences  &  Systems 
Washington,  D.  C.  20375 

EQS,  U.  S.  Marine  Corps 
ATTN:  CCA40  (Major  Pennell) 
Washington,  D.  C.  20380 

Commanding  Officer 
MCTSSA 

Marine  Corps  Base 

Camp  Pendleton,  CA  92055 

Chief,  C3  Division 
Development  Center 
MCDEC 

Quantico,  VA  22134 

Human  Factors  Technology  Administrator 

Office  of  Naval  Technology 

Code  MAT  0722 

800  N.  Quincy  Street 

Arlington,  VA  22217 

Commander 

Naval  Air  Systems  Command 
Human  Factors  Programs 
NAVAIR  334A 

Washington,  D.  C.  20361 


Department  of  the  Navy 
Commander 

Naval  Air  Systems  Command 
Crew  Station  Design 
NAVAIR  5313 

Washington,  D.  C.  20361 

Mr.  Philip  Andrews 
Naval  Sea  Systems  Command 
NAVSEA  03416 

Washington,  D.  C.  20362 
Commander 

Naval  Electronics  Systems  Cotsaand 
Human  Factors  Engineering  Branch 
Code  81323 

Washington,  D.  C.  20360 

Larry  Olmstead 

Naval  Surface  Weapons  Center 

NSWC/DL 

Code  N-32 

Dahlgren,  VA  22448 
Mr.  Milon  Essoglou 

Naval  Facilities  Engineering  Command 
RAD  Plans  and  Programs 
Code  03T 

Hoffman  Building  II 
Alexandria,  VA  22332 

Cape.  Robert  Biersner 
Naval  Medical  RAD  Command 
Code  44 

Naval  Medical  Center 
Bethesda,  MD  20014 

Dr.  Arthur  3achrach 
Behavioral  Sciences  Department 
Naval  Medical  Research  Institute 
Bethesda,  MD  20014 

Dr.  George  Moeller 
Human  Factors  Engineering  Branch 
Submarine  Medical  Research  Lab 
Naval  Submarine  Base 
Groton,  CT  06340 


22 


Department  of  the  Navy 
Read 

Aerospace  Psychology  Department 
Code  L5 

Naval  Aerospace  Medical  Research  Lab 
Pensacola ,  FL  32508 

Commanding  Officer 

Naval  Health  Research  Center 

San  Diego,  CA  92152 

Commander,  Naval  Air  Force, 

U,  S.  Pacific  Fleet 
ATTN:  Dr.  James  McGrath 
Naval  Air  Station,  North  Island 
San  Diego,  CA  92135 

Navy  Personnel  Research  and 
Development  Center 
Planning  A  Appraisal  Division 
San  Diego,  CA  92152 

Dr.  Robert  Blanchard 
Navy  Personnel  Research  and 
Development  Center 
Command  and  Support  Systems 
San  Diego,  CA  92152 

CDR  J.  Funaro 

Human  Factors  Engineelng  Division 
Naval  Air  Development  Center 
Warminster,  PA  18974 

Mr.  Stephen  Merriman 
Human  Factors  Engineering  Division 
Naval  Air  Development  Center 
Warminster,  PA  18974 

Mr.  Jeffrey  Grossman 
Hitman  Factors  Branch 
Code  3152 

Naval  Weapons  Center 
China  Lake,  CA  93555 

Human  Factors  Engineering  Branch 
Code  1226 

Pacific  Missile  Test  Center 
Point  Mugu,  CA  93042 


Department  of  the  Navy 

Dean  of  the  Academic  Departments 
U.  S.  Naval  Academy 
Annapolis,  MD  21402 

Dr.  S.  Schiflett 
Human  Factors  Section 
Systems  Engineering  Test 
Directorate 

0.  S.  Naval  Air  Test  Center 
Patuxent  River,  MD  20670 

Human  Factor  Engineering  Branch 
Naval  Ship  Research  and  Development 
Center,  Annapolis  Division 
Annapolis,  MD  21402 

Mr.  Harry.  Crisp 
Code  N  51 

Combat  Systems  Department 
Naval  Surface  Weapons  Center 
Dahlgren,  VA  22448 

Mr.  John  Quirk 

Naval  Coastal  Systems  Laboratory 
Code  712 

Panama  City,  FL  32401 

CDR  C.  Hutchins 
Code  55 

Naval  Postgraduate  School 
Monterey,  CA  93940 

Office  of  the  Chief  of  Naval 
Operations  (0P-115) 

Washington,  D.  C.  20350 

Professor  Douglas  E.  Hunter 
Defense  Intelligence  College 
Washington,  D.  C*  20374 

Department  of  the  Army 

Mr.  J.  Barber 

HQS,  Department  of  the  Army 
DAPE-MBR 

Washington,  D.  C.  20310 


23 


Department  of  Che  Saw 

Dr.  Edgar  M.  Johnson 
Technical  Director 
U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Director,  Organizations  and 
Systems  Research  Laboratory 
U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

Technical  Director 

U.  S.  Army  Human  Engineering  Labs 

Aberdeen  Proving  Ground,  MD  21005 

Department  of  the  Air  Force 

0.  S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NT. 

Soiling  Air  Force  Base 
Washington,  D.  C.  20332 

AFHRL/LRS  TDC 
Attn:  Susan  Ewing 
Wright-Patterson  AF3 ,  OH  45433 

Chief,  Systems  Engineering  Branch 
Human  Engineering  Division 
USAF  AMEL/HES 

Wright-Patterson  AFB,  OH  45433 

Dr.  Earl  Alluisi 
Chief  Scientist 
AFEEL/CCN 

Brooks  Air  Force  Base,  TX  78235 

Foreign  Addressees 

Dr.  Daniel  Kahneman 
University  of  British  Columbia 
Department  of  Psychology 
Vancouver,  BC  V6T  1W5 
Canada 


Foreign  Addressees 

Dr.  Kenneth  Gardner 
Applied  Psychology  Unit 
Admiralty  Marine  Technology 
Establishment 

Teddington,  Middlesex  TV11  0LN 
England 

Director,  Human  Factors  Wing 
Defence  &  Civil  Institute  of 
Environmental  Medicine 
Post  Office  Box  2000 
Downsview,  Ontario  M3M  339 
Canada 

Dr.  A.  D.  Baddeley 

Director,  Applied  Psychology  Unit 

Medical  Research  Council 

15  Chaucer  Road 

Cambridge,  CB2  2EF  England 

Other  Government  Agencies 

Defense  Technical  Information  Center 
Cameron  Station,  Bldg.  5 
Alexandria,  VA  22314  (12  copies)' 

Dr.  Craig  Fields 

Director,  System  Sciences  Office 
Defense  Advanced  Research  Projects 
Agency 

1400  Wilson  Blvd. 

Arlington,  VA  22209 

Dr.  M.  Montemerlo 
Human  Factors  &  Simulation 
Technology,  RTE-6 
NASA  HQS 

Washington,  D.  C.  20546 
Dr.  J.  Miller 

Florida  Institute  of  Oceanography 
University  of  South  Florida 
St.  Petersburg,  FL  33701 


24 


Other  Organizations 


Dr-  Robert  R.  Mackle 

Hunan  Factors  Research  Division 

Canyon  Research  Group 

5775  Dawson  Avenue 

Goleta,  CA  93017 

Dr.  Amos  Tversky 
Department  of  Psychology 
Stanford  University 
Stanford,  CA  94305 

Dr.  H.  Mcl.  Parsons 
Hunan  Resources  Research  Office 
300  N.  Washington  Street 
Alexandria,  VA  22314 

Dr.  Jesse  Orlansky 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  Street 
Alexandria,  VA  22311 

Professor  Howard  Ralffa 
Graduate  School  of  Business 
Administration 
Harvard  University 
Boston,  HA  02163 

Dr.  T.  B.  Sheridan 

Department  of  Mechanical  Engineering 
Massachusetts  Institute  of  Technology 
Cambridge,  MA  02139 

Dr.  Arthur  I.  Siegel 
Applied  Psychological  Services,  Inc. 
404  East  Lancaster  Street 
Wayne,  PA  19087 

Dr.  Paul  Slovic 
Decision  Research 
1201  Oak  Street 
Eugene,  OR  97401 

Dr.  Harry  Snyder 

Department  of  Industrial  Engineering 
Virginia  Polytechnic  Institute  and 
State  University 
Blacksburg,  VA  24061 


Ocher  Organizations 


Dr.  Ralph  Dusek 
Administrative  Officer 
Scientific  Affairs  Office 
American  Psychological  Association 
1200  17 ch  Street,  N.  W. 

Washington,  D.  C.  20036 

Dr.  Robert  I.  Eennessy 
NAS  -  National  Research  Council  (C0HF) 
2101  Constitution  Avenue,  N.  W. 
Washington,  D.  C.  20413 

Dr.  Amos  Freedy 
Perceptronics ,  Inc. 

6271  Variel  Avenue 
Woodland  Hills,  CA  91364 

Dr.  Robert  C.  Williges 
Department  of  Industrial  Engineering 
and  OR 

Virginia  Polytechnic  Institute  and 
State  University 
130  Whittemore  Hall 
Blacksburg,  VA  24Q61 

Dr.  Meredith  P.  Crawford 
American  Psychological  Association 
Office  of  Educational  Affairs 
1200  17th  Street,  N.  W. 

Washington,  D.  C.  20036 

Dr.  Deborah  Boehm-Davis 
General  Electric  Company 
Information  Systems  Programs 
1755  Jefferson  Davis  Highway 
Arlington,  VA  22202 

Dr.  Ward  Edwards 

Director,  Social  Science  Research 
Institute 

University  of  Southern  California 
Los  Angeles,  CA  90007 

Dr.  Robert  Fox 
Department  of  Psychology 
Vanderbilt  University 
Nashville,  TN  37240 


f 


25 


Ocher  Organizations 

Dr,  Charles  Gettys 
Department  of  Psychology 
University  of  Oklahoma 
455  West  Lindsey 
Norman,  OK  73069 

Dr,  Kenneth  Hammond 
Institute  of  Behavioral  Science 
University  of  Colorado 
Boulder,  CO  80309 

Dr,  James  H.  Howard,  Jr, 

Department  of  Psychology 
Catholic  University 
Washington,  D.  C.  20064 

Dr.  William  Howell 
Department  of  Psychology 
Rice  University 
Houston,  TX  77001 

Dr,  Christopher  Wickens 
Department  of  Psychology 
University  of  Illinois 
Urbana,  IL  61301 

Mr.  Edward  M.  Connelly 
Performance  Measurement 
Associates,  Inc. 

410  Pine  Street,  S.  E. 

Suite  300 
Vienna,  VA  22180 

Professor  Michael  Athans 
Room  35-406 

Massachusetts  Institute  of 
Technology 

Cambridge,  MA  02139 

Dr.  Edward  R.  Jones 
Chief,  Human  Factors  Engineering 
McDonnell-Douglas  Astronautics  Co. 
St.  Louis  Division 
Box  516 

St.  Louis,  M0  63166 


Other  Organizations 
Dr.  Babur  M.  Pulat 

Department  of  Industrial  Engineering 
North  Carolina  A6T  State  University 
Greensboro,  NC  27411 

Dr.  Lola  Lopes 

Information  Sciences  Division 
Department  of  Psychology 
University  of  Wisconsin 
Madison,  WI  53706 

Dr.  A.  K.  Bejczy 
Jet  Propulsion  Laboratory 
California'  Institute  of  Technology 
Pasadena,  CA  91125 

Dr.  Stanley  N.  Roscoe 
New  Mexico  State  University 
Box  5095 

Las  Cruces,  NM  88003 

• 

Mr.  Joseph  G.  Wohl 
Alphatech,  Inc. 

3  New  England  Executive  Park 
Burlington,  MA  01803 

Dr.  Marvin  Cohen 
Decision  Science  Consortium 
Suite  721 

7700  Leesburg  Pike 
Falls  Church,  VA  22043 

Dr.  Wayne  Zachary 
Analytics,  Inc. 

2500  Maryland  Road 
Willow  Grove,  PA  19090 

Dr.  William  R.  Uttal 
Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  MI  48109 

Dr.  William  B.  Rouse 
School  of  Industrial  and  Systems 
Engineering 

Georgia  Institute  of  Technology 
Atlanta,  GA  30332 


X  •>  .4 


tjktfcfc  'sklr,:  f  if?  *■* 


Ocher  Organizations 


Dr.  Richard  Pew 

Bole  Beranek  A  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02238 

Dr.  Hillel  Einhora 
Graduate  School  of  Business 
University  of  Chicago 
1 101  E.  58th  Street 
Chicago,  IL  60637 

Dr.  Douglas  Tovne 

University  of  Southern  California 

Behavioral  Technology  Laboratory 

3716  S.  Hope  Street 

Los  Angeles,  CA  90007 

Dr.  David  J.  Getty 

Bolt  Beranek  A  Newman,  Inc. 

50  Moulton  street 
Cambridge,  MA  02238 

Dr.  John  Payne 
Graduate  School  of  Business 
Admlnis  t ration 
Duke  University 
Durham,  NC  27706 

Dr.  Baruch  Fischhoff 
Decision  Research 
1201  Oak  Street 
Eugene,  OR  97401 

Dr.  Andrew  P.  Sage 
School  of  Engineering  and 
Applied  Science 
University  of  Virginia 
Charlottesville,  VA  22901 

Denise  Benel 
Essex  Corporation 
333  N.  Fairfax  Street 
Alexandria,  VA  22314 


